Programming Project - Unit 2

by Débora Azevedo, Eliseu Jayro, Francisco de Paiva and Igor Brandão

Goals The purpose of this project is explore the following:

  • Access Health Graph API - Runkeeper content;
  • Geolocation analysis and hypotheses should be explained in detail;

Wind Data Analysis for Sports

Como vimos no notebook “How long can you run”, o nosso amigo Ivano Vitch é um atleta exemplar que se preparou intensamente para a maratona de competições em sua cidade. Mas no decorrer de suas provas ele percebeu que existem fatores climáticos que influenciam direta ou indiretamente no seu treinamento. Agora o Ivano está focado em analisar o fator que mais impactou no seu desempenho durante as modalidades de exercícios que envolviam grandes deslocamentos na sua cidade; chamado: Vento. Esse fator é responsável por favorecer ou desfavorecer os resultados dos treinos dependendo se o seu sentido e velocidade estão a favor ou contra a trajetória tomada pelo atleta. Agora seguiremos analisando como poderemos ajudar o nosso amigo Vitsh a otimizar ainda mais os seus treinos e mostra-lo como ele poderá utilizar as informações da direção do vento na sua cidade para ampliar suas estratégias de treinamentos.

Contextualizing

Mesmo em uma rota relativamente montanhosa, a resistência do ar tem um efeito maior na velocidade geral do ciclismo do que o peso / resistência gravitacional. Quanto mais rápido o ciclista vai, mais resistência ao vento ele experimenta, e mais energia ele deve exercer para superá-lo.

  • A resistência do ar é tipicamente medida como CdA, , que é uma combinação da forma de um objeto (Cd) e sua área de superfície ( A ) Existem vários aspectos que podem interferir no CdA, como os que podem ser ligados a roupa do ciclista, formato da bicicleta ou mesmo a postura com que ele faz durante as pedaladas, podem gerar mais ou menos resistência diante do vento.

classificação

os resultados obtidos poderão ser divididos nas seguintes escalas de vento:

  • Normal Ruim Ótimo

Global Imports section

Import the necessary libraries to handle

  • Geocoding;
  • Maps;
  • File input;
  • Heatmap;
  • Numpy library;
  • Tqdm progress bar
In [ ]:
### Library necessary to run this IPython Notebook
!pip install geocoder
!pip install folium
!pip install tqdm
!pip install tabulate
!pip install pandas-datareader
!pip install requests
In [1]:
# Import pandas
import pandas as pd

# Import json
import json

# Import google geocoder
import geocoder as gc

# Import numpy library
import numpy as np

# Import folium maps
import os
import folium
from folium import plugins

# Import tqdm progressing bar plugin
from tqdm import tqdm

I - Maps section

Geolocation data import [optional]

In [2]:
# Import the geolocation.csv data: data
geolocation_data = pd.read_csv("geolocation.csv", encoding = 'latin2')

Coordinate groupping

The cell below perform the latitude/longitude reading and count all og them to know if there are repetitions and group [latitude, longitude, count] data into the coordinate array

In [ ]:
# =================================================================================
# Data adjusts
# =================================================================================

# Count the same latitude/longitude occurrences
geodata = geolocation_data.groupby(['longitude', 'latitude']).size().reset_index().rename(columns={0:'count'})

# =================================================================================
# Add the coordinates
# =================================================================================

# Initialize the coordinates array
coordinates = []

# Add the coordinates to the coordinate
for i in tqdm(range(len(geodata))):
    # eliminate items with'nan' element
    if all(~np.isnan([geodata.loc[geodata.index[i],'latitude'], geodata.loc[geodata.index[i],'longitude']])):
        coordinates.append([geodata.loc[geodata.index[i],'latitude'], geodata.loc[geodata.index[i],'longitude'], 
                            geodata.loc[geodata.index[i],'count']])

# Display the imported coordinates
coordinates

Add the coordinates to the maps [required]

In [ ]:
# =================================================================================
# Map settings
# =================================================================================

# Set map center and zoom level
mapc = [-5.788, -35.202]
zoom = 11

# Wind rose
url = ('https://raw.githubusercontent.com/SECOORA/static_assets/'
       'master/maps/img/rose.png')

# Create HeatMap instance
htMap = folium.Map(location=mapc, zoom_start=zoom, tiles='OpenStreetMap', control_scale=True)
htMap2 = folium.Map(location=mapc, zoom_start=zoom, tiles='OpenStreetMap', control_scale=True)

# Create marker map instance
mkMap = folium.Map(location=mapc, zoom_start=zoom, tiles='OpenStreetMap', control_scale=True)

# Create circle marker map instance
circMap = folium.Map(location=mapc, zoom_start=zoom, tiles='OpenStreetMap', control_scale=True)

# Chunk size
chunksize = 10 ** 3 # can be changed

# For item i in a range that is a length of l,
for i in tqdm(range(0, len(coordinates), chunksize)):
    # Append the coordinates to the Heat Map
    plugins.HeatMap(coordinates[i:i+chunksize]).add_to(htMap)
    plugins.FloatImage(url, bottom=5, left=75).add_to(htMap)
    
    plugins.HeatMap(coordinates[i:i+chunksize]).add_to(htMap2)
    plugins.FloatImage(url, bottom=5, left=75).add_to(htMap2)
    
    # Append the coordinates to Marker Map
    plugins.MarkerCluster(coordinates[i:i+chunksize]).add_to(mkMap)
    plugins.FloatImage(url, bottom=5, left=75).add_to(mkMap)

1) Heat Map

In [5]:
# print the map
htMap
Out[5]:

2) Cluster Map

In [29]:
# print the map
mkMap
Out[29]:

3) With Natal Neighborhoods

In [7]:
# import geojson file about natal neighborhoods
natal_neigh = os.path.join('geojson', 'natal.geojson')

# load the data and use 'UTF-8'encoding
geo_json_natal = json.load(open(natal_neigh))
In [ ]:
# print the keys of the dictionary
print(geo_json_natal.keys())
# print the list of features (neighborhoods)
geo_json_natal['features']
In [9]:
neighborhood = []
# list all neighborhoods
for neigh in geo_json_natal['features']:
        neighborhood.append(neigh['properties']['name'])

3.1) Heatmap with Natal neighborhoods

In [10]:
# Create new map instance
htMapNeighborhood = htMap2

# Configure geojson layer
folium.GeoJson(geo_json_natal).add_to(htMapNeighborhood)

# Add map layers
folium.TileLayer('OpenStreetMap').add_to(htMapNeighborhood)
folium.TileLayer('Mapbox Bright').add_to(htMapNeighborhood)
folium.TileLayer('stamentoner').add_to(htMapNeighborhood)
folium.TileLayer('openstreetmap').add_to(htMapNeighborhood)
folium.TileLayer('Mapbox Bright').add_to(htMapNeighborhood)
folium.TileLayer('Mapbox Control Room').add_to(htMapNeighborhood)
folium.TileLayer('stamenterrain').add_to(htMapNeighborhood)
folium.TileLayer('stamentoner').add_to(htMapNeighborhood)
folium.TileLayer('stamenwatercolor').add_to(htMapNeighborhood)
folium.TileLayer('cartodbpositron').add_to(htMapNeighborhood)
folium.TileLayer('cartodbdark_matter').add_to(htMapNeighborhood)

folium.LayerControl().add_to(htMapNeighborhood)
               
# print the map
htMapNeighborhood
Out[10]:

3.2) Cluster map with Natal neighborhoods

In [ ]:
# Create new map instance
mkMapNeighborhood = mkMap

# Configure geojson layer
folium.GeoJson(geo_json_natal).add_to(mkMapNeighborhood)

# print the map
mkMapNeighborhood

4) Choropleth map

Use this cell if the neighborhood frequency already has been processed

In [23]:
neighborFrequency = pd.read_csv("neighborFrequency.csv", encoding = 'latin2')

# Group the neighbor data
neighborFrequency = neighborFrequency.groupby(['district']).size().reset_index().rename(columns={0:'count'})

Service to convert lat/lng to neighborhood [It consumes external API - too heavy!]

Just execute this cell if the file neighborFrequency.csv wasn't generated before

In [ ]:
# =================================================================================
# Coordinate reversing
#
# convert lat/lng to neighborhood
# =================================================================================
import requests
import json
from pandas.io.json import json_normalize

# Access token
API_KEY = '574ad2a81d281f'

# Function to get the neighborhood from lat/lng
def getplace(lat, lng):
    # Base URI
    url = "https://us1.locationiq.com/v1/reverse.php"

    # Parameter data
    data = {
        'key': API_KEY,
        'lat': lat,
        'lon': lng,
        'format': 'json'
    }

    # Perform the request
    response = requests.get(url, params=data).json()
    
    # Check the return
    if 'address' in response:
        return response['address']['suburb']
    else:
        return ''

# =================================================================================

# Copy the data
neighborFrequency = pd.DataFrame()

# Set a limit
LIMIT = len(coordinates)
# LIMIT = 100

# Perform a massive convertion from lat/lng to suburb
for i in tqdm(range(LIMIT)):
    # Call the conversion function
    itemData = {
        'district': getplace(coordinates[i][0], coordinates[i][1]),
        'count': 1,
    }

    # Convert the data to json
    itemData = json_normalize(itemData)

    # Add the data to neighborhood dataFrame
    neighborFrequency = pd.concat([neighborFrequency, itemData])

# Group the neighbor data
neighborFrequency = neighborFrequency.groupby(['district']).size().reset_index().rename(columns={0:'count'})

# Export the neighbor data to csv
neighborFrequency.to_csv('neighborFrequency.csv', encoding="utf-8")

Choropleth map color scale

In [24]:
from branca.colormap import linear

colormap = linear.YlGn_09.scale(
    neighborFrequency['count'].min(),
    neighborFrequency['count'].max())

print(colormap(1.0))

colormap

#ffffe5
Out[24]:
4613
In [25]:
neighbor_dict = neighborFrequency.set_index('district')['count']

# Function to get the neighborhood from lat/lng
def applyColor(district):
    try:
        return colormap(neighbor_dict[district])
    except KeyError:
        return '#ffffe5'
    
# Function to get the district count
def districtCount(district):
    try:
        return neighbor_dict[district]
    except KeyError:
        return 0
    
def highlight_function(district):
    return {
        'fillColor': '#ffaf00',
        'color': 'green',
        'weight': 3,
        'dashArray': '5, 5',
    }
In [26]:
# =================================================================================
# Map settings
# =================================================================================

# Set map center and zoom level
mapc = [-5.788, -35.202]
zoom = 11

# definition of the boundaries in the map
geo_json_data = json.load(open(natal_neigh))

# creation of the choropleth
choroplethNatal = folium.Map(location=mapc, zoom_start=zoom, tiles='OpenStreetMap', control_scale=True)

# Wind rose
url = ('https://raw.githubusercontent.com/SECOORA/static_assets/'
       'master/maps/img/rose.png')

# set the map data
folium.GeoJson(
    geo_json_data,
    name='exercise data',
    style_function=lambda feature: {
        'fillColor': applyColor(feature['properties']['name']),
        'key_on': 'feature.properties.name',
        'color': 'black',
        'weight': 1,
        'dashArray': '5, 5',
        'fillOpacity': 0.7,
        'legend_name': 'Exercises in Natal',
        'highlight': True,
    },
    highlight_function=lambda feature: highlight_function(feature['properties']['name']),
    tooltip=folium.GeoJsonTooltip(fields=['name'], aliases=['Neighborhood'])
).add_to(choroplethNatal)

# Circle on map
popup = 'Current location (Instituto Metropole Digital - IMD)'

folium.CircleMarker(
    location=[-5.832190, -35.205319],
    radius=20,
    fill=True,
    popup=popup,
    weight=1,
).add_to(choroplethNatal)

# Add the map legend
colormap.caption = 'Neighborhoods exercise frequency (absolute numbers)'
colormap.add_to(choroplethNatal)

# Wind rose on map
plugins.FloatImage(url, bottom=5, left=75).add_to(choroplethNatal)

# Add map layers
folium.TileLayer('OpenStreetMap').add_to(choroplethNatal)
folium.TileLayer('Mapbox Bright').add_to(choroplethNatal)
folium.TileLayer('stamentoner').add_to(choroplethNatal)
folium.TileLayer('openstreetmap').add_to(choroplethNatal)
folium.TileLayer('Mapbox Bright').add_to(choroplethNatal)
folium.TileLayer('Mapbox Control Room').add_to(choroplethNatal)
folium.TileLayer('stamenterrain').add_to(choroplethNatal)
folium.TileLayer('stamentoner').add_to(choroplethNatal)
folium.TileLayer('stamenwatercolor').add_to(choroplethNatal)
folium.TileLayer('cartodbpositron').add_to(choroplethNatal)
folium.TileLayer('cartodbdark_matter').add_to(choroplethNatal)

folium.LayerControl().add_to(choroplethNatal)
               
# print the map
choroplethNatal
Out[26]:
In [ ]: